Dynamic adaptation to increased sfu load by disabling video streams

ABSTRACT

A method for dynamically adapting to increased system load at a selective forwarding unit (SFU) is disclosed. In an embodiment, an SFU sends a plurality of video streams to a plurality of participant computing devices. The SFU monitors a system load value on the SFU according to any of a plurality of metrics. When the SFU determines that the monitored system load value exceeds a stored load threshold value, the SFU selects one or more of the streams being sent to participant computing devices and disables them.

BENEFIT CLAIM

This application claims the benefit under 35 U.S.C. § 120 as aContinuation of application Ser. No. 15/493, 057, which claims thebenefit under 35 U.S.C. § 120 as a Continuation of application Ser. No.15/279,325, filed Sep. 28, 2016, the entire contents of both of whichare hereby incorporated by reference for all purposes as if fully setforth herein. The applicants hereby rescind any disclaimer of claimscope in the parent applications or the prosecution history thereof andadvise the USPTO that the claims in this application may be broader thanany claim in the parent applications.

TECHNICAL FIELD

The technical field of the present disclosure is computer hardware,software and systems that implement audio and video conferencing. Thetechnical field of the disclosure also is computer hardware, softwareand systems that are programmed for detecting and dynamically adaptingto overuse and underuse of system resources in Selective ForwardingUnits (SFU).

BACKGROUND

The approaches described in this section could be pursued, but are notnecessarily approaches that have been previously conceived or pursued.Therefore, unless otherwise indicated herein, the approaches describedin this section are not prior art to the claims in this application andare not admitted to be prior art by inclusion in this section.

Video and audio calls and conferences are becoming both more common andlarger in size every day. Originally, video and audio calls andconferences hosted using computers were limited to direct connectionsbetween users in which a first user called a second user, enabling theusers to see and hear each other during the call. As methods improved,conferences of five, ten, or more users all with streaming video andaudio became more common. Simply sending all video and audio data fromeach participant to every other participant in such a large conferenceoften is not feasible due to limitations on network bandwidth in localarea networks and internetworks and the limited processing power in someclient devices.

In response, solutions to large video and audio conferences wereimplemented with server computers using two general approaches:multipoint control units (MCU) and selective forwarding units (SFU).MCUs receive and process audio and video data from clients and typicallymix the audio and video to create a single composite video. Thecomposite video is then streamed to the clients in a single mix. Whilethis solution enables relatively simple client software, the processingpower needed for the MCU is high, and the resulting presentation at theclient is generally inflexible. SFUs, in contrast, are more similar topacket data routers, as SFUs forward packets without media processing.

Due to limitations on network bandwidth and processing power, SFUs mayimprove their efficiency by only sending a subset of the video streamsto each computing device. In an extremely large video conference,sending all of the video streams to each of the computing devices in theconference would be egregious, as most of the video streams are likelyunnecessary to the majority of the people in the video conference. Thus,the SFU may reduce the load on the SFU by limiting the number of videostreams sent to each participant computing device in a conference.

While setting a hard limit on the number of video streams sent to eachcomputing device in a conference effectively reduces the load on theSFU, a hard limit is not adaptable to different situations. Forinstance, there may be times when the load on the SFU is especiallyhigh, such as during peak conferencing hours or in the case of aparticularly large conference. When the SFU begins overusing itsavailable resources, the ability of the SFU to forward high qualityvideos begins to degrade. For example, when the SFU uses too muchbandwidth, the SFU begins to lose packets over the network, resulting inbad video quality for all users of the SFU.

A solution to the overuse of network resources is to create strongerlimits on network usage, such as by using a smaller limit on the numberof video streams sent to each participant computing device in aconference. While this method would reduce the usage of networkresources, it would also degrade the experience for members of theconference. In times of low system usage, each participant computingdevice would still be limited on the number of received video streams,even though the SFU has bandwidth to send a greater number of videostreams.

Thus, there is a need for a method that allows an SFU to dynamicallyadapt to overuse and underuse of system resources while minimizing thenegative effects on user experience.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIG. 1 illustrates an example computer system in which the techniquesdescribed may be practiced, according to one embodiment.

FIG. 2 illustrates a programmable algorithm or method in accordance withan embodiment.

FIG. 3 illustrates an example implementation of selecting participantcomputing devices by selecting one or more conference sets and reducingthe maximum stream value for the selected conference sets.

FIG. 4 illustrates a computer system upon which an embodiment may beimplemented.

DESCRIPTION OF EXAMPLE EMBODIMENTS

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

General Overview

Computer-implemented techniques are provided for dynamically adapting tochanges in system load at a selective forwarding unit (SFU).Specifically, in various programmable algorithms, an SFU monitors asystem load on the SFU. When the SFU determines that it is overusingavailable resources, the SFU selectively disables streams in a mannerthat reduces the system load while minimizing the negative effect on theusers. In an embodiment, streams are selected such that disabling thestreams reduces the system load past an overuse threshold. The streamsmay further be selected in a manner that reduces a participant effectvalue which estimates an effect on the participants of disabling thestreams.

In one embodiment, a method comprises sending, from a selectiveforwarding unit (SFU) to each of a plurality of participant computingdevices, one or more digital video streams, wherein the SFU storesidentifiers of each participant computing device among the plurality ofparticipant computing devices; monitoring, at the SFU, a system loadvalue which estimates a load on the SFU according to any of a pluralityof different load metrics; determining that the system load valueexceeds a particular load threshold value and, in response, selecting asubset of the identifiers of the plurality of participant computingdevices; for each participant computing device that corresponds to oneof the identifiers of the subset, disabling one or more of the digitalvideo streams that are sent to the participant computing device from theSFU.

In order to adapt to underuse, in one embodiment, the method furthercomprises, in response to disabling one or more of the digital videostreams that are sent to the participant computing device of the subsetof the plurality of participant computing devices, waiting a particularperiod of time; after waiting the particular period of time, determiningthat the system load value is below a second load threshold value and,in response, selecting one or more participant computing devices of thesubset of the plurality of participant computing devices; restoring oneor more of the disabled digital video streams.

Example System Implementation

FIG. 1 illustrates an example computer system in which the techniquesdescribed may be practiced, according to one embodiment. In anembodiment, a computer system 100 comprises components that areimplemented at least partially by hardware at one or more computingdevices, such as one or more hardware processors executing storedprogram instructions stored in one or more memories for performing thefunctions that are described herein. In other words, all functionsdescribed herein are intended to indicate operations that are performedusing programming in a special-purpose computer or general-purposecomputer, in various embodiments. System 100 illustrates only one ofmany possible arrangements of components configured to execute theprogramming described herein. Other arrangements may include fewer ordifferent components, and the division of work between the componentsmay vary depending on the arrangement.

System 100 includes selective forwarding unit (SFU) 105 and participantcomputing devices 130, 140, and 150, which may be communicativelycoupled directly or indirectly to the SFU via networks. SFU 105 may beany computing device, including but not limited to: servers, racks, workstations, personal computers, general purpose computers, laptops,Internet appliances, hand-held devices, wireless devices, wired devices,portable or mobile devices, wearable computers, cellular or mobilephones, portable digital assistants (PDAs), smart phones, tablets,multi-processor systems, microprocessor-based or programmable consumerelectronics, game consoles, set-top boxes, network PCs, mini-computers,virtual machines, and the like. Although FIG. 1 shows SFU 105 as asingle device, SFU 105 may be multiple devices, such as a servercluster, and may be located in one or more physical locations. Asdiscussed above, a SFU forwards audio and video packets for a conferencewithout media processing.

SFU 105 is communicatively connected to client devices 130, 140 and 150through any kind of computer network using any combination of wired andwireless communication, including, but not limited to: a Local AreaNetwork (LAN), a Wide Area Network (WAN), the Internet, a companynetwork, etc. SFU 105 may include applications, software, and otherexecutable instructions to facilitate various aspects of embodimentsdescribed herein. In one embodiment call bridge 110 comprises programinstructions that are configured to receive RTP packets, selectparticipants for load adaptation, monitor the load of the SFU, disablestreams being sent to one or more participant computing devices,determine bit-rates for particular streams, selectively forward RTPpackets based on a variety of factors, and any other functions needed toimplement the present disclosure. In one embodiment, SFU 105 comprisesparticipant selection instructions 104, SFU load monitoring instructions106, and load adaptation instructions 108, each coupled to a call bridge110. The participant selection instructions 104, SFU load monitoringinstructions 106, and load adaptation instructions 108 are stored withinelectronic digital storage as indicated by main memory 102. Each of theparticipant selection instructions 104, SFU load monitoring instructions106, and load adaptation instructions 108 may comprise programmedmachine executable instructions that are configured or programmed toimplement and execute the algorithms that are further described herein,including in the flow diagrams and the text relating to those diagrams.In various embodiments, the participant selection instructions 104, SFUload monitoring instructions 106, and load adaptation instructions 108may be created and stored using a human-readable source code languageand using a program development environment including but not limited toJAVA, OBJECTIVE-C, C++, C and the like.

In one embodiment, call bridge 110 comprises program instructions suchas participant selection instructions 104 that may be stored digitallyin memory and configured to, when executed by one or more processors,cause call bridge 110 to identify particular participant computingdevices or particular groups of computing devices for load adaptation.Specifically, in embodiments where participant computing devices arereceiving a plurality of streams, execution of participant selectioninstructions 104 may cause the call bridge 110 to identify participantcomputing devices which are receiving a plurality of streams and reducethe number of streams for the participant computing devices.

In one embodiment, call bridge 110 comprises program instructions suchas SFU load monitoring instructions 106 that may be stored digitally inmemory and configured to, when executed by one or more processors, causecall bridge 110 to measure a load on the SFU and determine whether theload on the SFU is outside specified parameters. Specifically, executingSFU load monitoring instructions 106 may cause the call bridge 110 todetermine whether a load on the SFU is greater than a particular overusethreshold. Additionally, executing SFU load monitoring instructions maycause the call bridge 110 to determine whether a load on the SFU is lessthan a particular underuse threshold.

In one embodiment, call bridge 110 comprises program instructions suchas load adaptation instructions 108 that may be stored digitally inmemory and configured to, when executed by one or more processors, causecall bridge 110 to alter outgoing streams to participants in order tobalance the load on SFU 105. Specifically, executing the load adaptationinstructions 108 may cause call bridge 110 to selectively decrease anumber of outgoing streams and/or selectively lower the quality ofoutgoing streams in response to a determination that a load on the SFUis greater than the particular overuse threshold. Additionally,executing load adaptation instructions 108 may cause call bridge 110 toselectively increase a number of outgoing streams and/or selectivelyincrease the quality of outgoing streams in response to a determinationthat a load on the SFU is lower that the particular underuse threshold.

For purposes of illustrating a clear example, FIG. 1 shows threeparticipant computing devices 130, 140, 150, but in other embodiments,any number of client devices may be used and the embodiments hereinspecifically contemplate interoperation with thousands or millions ofclient devices with dozens or more client devices participating in anyone audio conference or video conference. In one embodiment, theparticipant computing devices 130, 140, 150 are computing devices,including but not limited to: work stations, personal computers, generalpurpose computers, laptops, Internet appliances, hand-held devices,wireless devices, wired devices, portable or mobile devices, wearablecomputers, cellular or mobile phones, portable digital assistants(PDAs), smart phones, tablets, multi-processor systems,microprocessor-based or programmable consumer electronics, gameconsoles, set-top boxes, network PCs, mini-computers, servers, racks,and the like.

Participant computing devices 130, 140, 150 are communicativelyconnected to SFU 105 using any combination of wired and wirelesscommunication as represented by the arrows in FIG. 1, including, but notlimited to: a Local Area Network (LAN), a Wide Area Network (WAN), theInternet, a company network, etc. Participant computing devices 130,140, 150 may include applications, software, and other executableinstructions to facilitate various aspects of embodiments describedherein. Specifically, participant computing devices 130, 140, 150 mayinclude client software that is programmed to support video conferencingor audio conferencing. The software may be standalone, may beimplemented through a web browser using browser-executable code, or anyother suitable application.

In an embodiment, each of participant computing devices 130, 140, 150are part of conference set 120. A conference set, as used herein, refersto a plurality of computing devices which are identified asparticipating in the same conference. Video and audio streams receivedfrom participant computing devices of the same conference may beforwarded to other participant computing devices of the same conference.Different participant computing devices in a conference set may receivedifferent video and audio streams. For example, a user of a firstparticipant computing device may request a video feed from a differentparticipant computing device of a conference set than a user of a secondparticipant computing device. Thus, while both computing devices receivevideo streams from other computing devices in the conference set, thereceived video streams may differ. Additionally, a particularparticipant computing device may not receive video and/or audio streamsthat the particular participant computing device sends to SFU 105.

In an embodiment, one or more participant computing devices ofconference set 120 receive a plurality of video and/or audio streams.For example, SFU 105 may forward video and audio streams from multiplecomputing devices of the conference set to a single computing device ofthe conference set. In a video conference setting, this allows aparticipant computing device to display multiple video and/or audiostreams from a video conference where multiple participant computingdevices are generating and forwarding video and audio simultaneously.While some participant computing devices may receive a plurality ofvideo and/or audio streams, other participant computing devices mayreceive one or zero video and/or audio streams.

SFU 105 may be programmed or configured to identify a subset of receivedvideo and/or audio streams that SFU 105 receives from devices in aparticular conference set to send to other participant computing devicesin the conference set. For example, SFU 105 may, for each conferenceset, store a maximum stream value indicating a maximum number of streamsto send to each computing device of the conference set. Thus, if aparticular conference set includes thirty participant computing devicesbut the maximum number of streams for the particular conference set isfive, then each participant computing device may receive a maximum offive streams from SFU 105.

SFU 105 may select the subset of streams from the plurality of streamsreceived from the participant computing devices of the particularconference set to send to each participant computing device based on oneor more factors. Factors may include selection by participant computingdevices and/or audio levels of the received streams. For example, SFU105 may be configured to identify five streams each of which forwardedhigher than average audio levels for that stream more recently than theremaining streams. Thus, streams that included the most recent speakersmay be forwarded to participant computing devices of a particularconference set. As different streams are selected to be included in thesubset of streams, SFU may remove streams being sent to participantcomputing devices and replace them with the newly selected streams. Forexample, if a particular participant computing device is selected forthe subset due to a monologue of a participant, SFU 105 may send streamsfrom that particular participant computing device to other participantcomputing devices of the conference set. If after a period of time, fiveother participant computing devices are selected to be in the subset,then SFU 105 may stop sending streams from the particular participantcomputing device in favor of the five other participant computingdevices.

In an embodiment, each participant computing device may additionallystore a maximum stream value indicating a maximum number of streams thatthe participant computing device may receive. For example, a firstparticipant computing device may have greater capabilities than a secondparticipant computing device. Thus, the first participant computingdevice may be capable of receiving more video streams than the secondparticipant computing device. The stored maximum values may change overtime. For example, a particular participant computing device may monitorits internal CPU load and increase or decrease the stored maximum streamvalue based on the internal CPU load. Each participant computing devicemay send the stored value indicating the maximum number of streams thatthe participant computing device may receive to SFU 105. SFU 105 mayselect, for each participant computing device, a number of streams tosend to the participant computing device that is less than or equal tothe lesser of the SFU's maximum stream value and the participantcomputing device's maximum stream value.

Load Adaptation

FIG. 2 illustrates a programmable algorithm or method in accordance withan embodiment. FIG. 2 is intended to describe an algorithm that could beprogrammed in the system described above at the same level of detailthat persons of skill in the art to which this invention pertains willnormally use to communicate among themselves with respect to otheralgorithms, methods, program plans and the like. Persons of this levelof skill are accustomed to communicating at the level of FIG. 2 andpossess the education, information and skill that can be applied to FIG.2 to result in completing computer programs that execute to practice thefunctions shown in FIG. 2 and further described herein.

At step 202, one or more video streams are sent to each of a pluralityof participant computing devices from a selective forwarding unit (SFU).The SFU stores identifiers of each participant computing device amongthe plurality of participant computing devices. For example, a pluralityof participant computing devices in a particular conference set may eachsend video streams to the SFU. The SFU may then, for one or moreparticipant computing devices in the conference set, select a subset ofthe received video streams to forward to the participant computingdevice.

SFU 105 may store, for each participant computing device, an identifierof the participant computing device, such as a username. SFU 105 mayalso store conference set identifiers which identify particulargroupings of participant computing devices that are participating in aparticular conference. For example, a particular conference setidentifier may be associated with a plurality participant computingdevice identifiers in memory of SFU 105. Additionally, SFU 105 may storedata which identifies, for each identifier of a participant computingdevice, a number of streams received by the participant computingdevice, a quality of streams received by the participant computingdevice, and/or one or more capabilities of the participant computingdevice.

At step 204, a system load value which estimates a load on the SFUaccording to any of a plurality of metrics is monitored at the SFU. Thesystem load value may be any of network utilization, CPU usage, outgoingbitrate, physical memory usage, other metrics for measuring load on asystem, and/or any combination of the aforementioned metrics. Forexample, SFU 105 may monitor both the CPU usage of SFU 105 and thebandwidth usage of SFU 105 to determine if either value exceeds presetlimits. In some embodiment, to monitor the metrics, the SFU 105 mayretrieve data from stored sources, such as simple network managementprotocol (SNMP) management information base (MIB) objects.Alternatively, one program or method that implements part of FIG. 2 maysend a programmatic call or request to another method, object or programto retrieve metrics that a separate thread or process is monitoring. Thespecific method of monitoring load metrics is not critical.

At step 206, a determination is made that the system load value exceedsa particular load threshold value and, in response, a subset of theidentifiers of the plurality of participant computing devices isselected. SFU 105 may store, in memory or persistent storage, an overuseload threshold value for the monitored system load value whichidentifies a value above which SFU 105 determines that SFU 105 isoverusing available resources. For example, an overuse load thresholdvalue for network utilization, measured as average egress bitrate on theserver over the last five seconds, may be set to 200 Mbps. Thus, if theaverage egress bitrate on the server over the last five seconds exceeds200 Mbps, SFU 105 may determine that the system is overusing resources.

The overuse load threshold value may also comprise a percentage ofavailable resources that are being used by SFU 105. For example, SFU 105may be programmed or configured to monitor a percentage of available CPUbeing used as the system load value. The overuse load threshold valuemay be set as a particular percentage of the available CPU, such as 90%.If SFU 105 determines that more than 90% of the available CPU is beingused, SFU 105 may determine that the system load value exceeds thethreshold. By setting the overuse load threshold value at a percentageof available resources, the system and method described herein may beimplemented on different SFUs with different capabilities without thesetting of new thresholds based on those capabilities. Additionallyand/or alternatively, the overuse load thresholds may be calculateddynamically, such as through estimations of the effects on the CPU of acurrent CPU load. For example, the overuse load threshold may becomputed based on a total number of dropped packets over a particularperiod of time.

SFU 105 may be programmed or configured to select the subset of theplurality of participant computing devices such that disabling one ormore streams for the subset of the plurality of participant computingdevices would cause the system load value to decrease past a particularvalue. For example, if each stream includes a constant bitrate of 2Mbps, in order to lower the system load value by 20 Mbps, SFU 105 mayselect participant computing devices to be part of the subset such thatten streams total are disabled.

SFU 105 may be programmed or configured to reduce the system load valuebelow the overuse load threshold value. Thus, in an embodiment, theparticular value is the overuse load threshold value. In otherembodiments, the particular value is a value below the overuse loadthreshold value. For example, if the overuse load threshold value is 200Mbps, SFU 105 may be programmed or configured to select the subset suchthat disabling the streams for the subset brings the system load valuebelow 200 Mbps, the overuse load threshold, or below a safety value,such as 190 Mbps.

In an embodiment, SFU 105 selects the subset of the plurality ofparticipant computing devices such that disabling streams for the subsetof the plurality of participant computing devices minimizes aparticipant effect value. The participant effect value, as used herein,refers to an estimated effect on the participant computing devices ofdisabling one or more streams for the participant computing devices. Forexample, for each participant computing device, an effect on theparticipant computing device of disabling x streams may be computed asx/n where n is the total number of streams being received by theparticipant computing device. Thus, if a participant computing device isreceiving five streams, the effect on the participant computing deviceof disabling one of the five streams would be ⅕. The participant effectvalue may comprise a summation of the individual effects on theparticipant computing devices.

In an embodiment, SFU 105 uses mathematical minimization techniques toidentify the subset of the plurality of participant computing devicessuch that the system load value is decreased below the overuse loadthreshold value while minimizing the participant effect value. In simplescenarios where all streams are assumed to comprise the same bitrate,selecting the subset of participant computing devices may compriseselecting the participant computing devices with the highest number ofstreams, thereby minimizing the participant effect value for the samedecrease in the system load value.

In more complex situations, where different streams comprise differentbitrates, SFU may perform more complex minimizations. For example, if afirst participant computing device receives five streams at 3 Mbps and asecond participant computing device receives four streams at 4 Mbps,selecting the second participant computing device instead of the firstparticipant computing device would increase the effect on the systemload value, but at the cost of increasing the participant effect value.Thus a selection of the second participant computing device over thefirst may depend on the overuse load threshold value, the bitrates ofstreams of other participant computing devices, and/or the number ofstreams of the other participant computing devices.

In an embodiment, an equation for selecting a subset, A, of theparticipant computing devices of SFU 105, B, comprises:

${{A\left( {{P\left( {r,n} \right)},x} \right)} \subseteq {B(P)}}\begin{pmatrix}{{\sum\limits_{i}^{A}\; {r_{i}x_{i}}} \geq {L - T_{L}}} \\{\min \left( {\sum\limits_{i}^{A}\frac{x_{i}}{n_{i}}} \right)}\end{pmatrix}$

where P_(i)(r_(i),n_(i)) is a participant computing device receivingn_(i) streams with a bitrate of r_(i) and being selected to have x_(i)streams disabled. L is the current system load value and T_(L) is theoveruse load threshold value. |A| is the number of selected participantcomputing devices in the subset A.

Conference Set Maximum Stream Values

In an embodiment, SFU 105 selects the subset of participant computingdevices by selecting particular conference sets and lowering the maximumstream value for the conference set. SFU 105 may compute, for eachconference set, a participant effect value and an effect on the systemload value based on a reduction of the maximum stream value for theconference set. SFU 105 may select conference sets for lowering amaximum stream value by a particular number of streams such that thesystem load value is reduced past the particular value and theaggregated participant effect value is minimized.

FIG. 3 illustrates an example implementation of selecting participantcomputing devices by selecting one or more conference sets and reducingthe maximum stream value for the selected conference sets. The exampleillustrated in FIG. 3 depicts the lowering of a maximum stream value fora conference set by one. In an embodiment, an SFU may be configured tolower the maximum stream value for a conference by a different value,based, at least in part, on an effect on the user and/or the currentsystem load.

FIG. 3 includes SFU 302 communicatively coupled to conference sets 304,306, 308, and 310. SFU 302 has a maximum overuse load threshold value of200 Mbps and a current system load value of 210 Mbps. Thus, SFU 302 willattempt to identify one or more conference sets that reduce the currentsystem load value by 10 Mbps or more while minimizing a participanteffect value. In the example of FIG. 3, each stream is assumed to be aconstant bitrate of 2 Mbps for the purpose of simplicity. In morecomplex embodiments, the load effect and participant effect value foreach conference are computed as sums of individual load effects andparticipant effect values for each participant computing deviceaffected.

Of the conference sets in FIG. 3, lowering the maximum stream value forconference set 306 generates the lowest participant effect value of 2while decreasing the current load past the maximum load. Therefore, inthe example of FIG. 3, SFU 302 would select conference set 306 to havethe maximum stream value lowered.

Conference set 304 contains 20 participants and has a maximum streamvalue of 3. Thus, at a constant bitrate of 2 Mbps per stream, the loadeffect is 40 Mbps while the participant effect is 6.66. While the loadeffect satisfies the first requirement, that the load effect is 10 Mpbsor greater, the participant effect of 6.66 is higher for conference set304 than any of the other conference sets.

Conference set 308 has the lowest participant effect, but the loadeffect of 8 Mbps is lower than 10 Mbps. While conference set 308 couldbe combined with a different conference set to reach the 10 Mbps loadreduction, no combination with the available remaining conference setsof FIG. 3 would minimize the participant effect value. If SFU 105lowered the maximum stream value for conference set 308 by two, the loadeffect would be doubled to 16 Mbps, which would decrease the currentload past the maximum load. The participant effect of decreasing themaximum stream value for conference set 308 by two would be 4×(2/3), or2.66. As the participant effect of decreasing the maximum stream valuefor conference set 308 by two is greater than the participant effect ofdecreasing the maximum stream value for conference set 306 by one, SFU302 would not select conference set 308 over conference set 306.

Conference set 310 has the highest maximum stream value of theconference sets in FIG. 3. In embodiments where participant computingdevices are selected individually, SFU 302 would select either five orsix of the participant computing devices in conference set 310 in orderto minimize the participant effect value. In embodiments where themaximum stream value is lowered by conference set, the larger number ofparticipants in conference set 310 causes the participant effect valueto be higher than the participant effect value for conference set 306.

The implementation depicted in FIG. 3 assumes that each participant isreceiving the maximum number of streams. In embodiments, a participantcomputing devices in a conference set can store a maximum stream valuefor itself which is lower than the current maximum stream value for theconference set. Thus, in order to determine the load effect and theparticipant effect values, SFU 302 may first determine how manyparticipant computing devices would be affected by reducing the maximumstream value for the conference. As an example, if, in FIG. 3, only fourof the participants of conference set 306 were receiving five streams,then the load effect of lowering the maximum stream value of onlyconference set 306 by one would not be large enough to reduce thecurrent system load value to the overuse load threshold value. Thus, foreach conference set, SFU 302 may be configured to compute the actualload effect and participant effect values when the maximum stream valueis lowered for the conference set based on a subset of the participantcomputing devices of the conference set that would be affected by thelowered maximum stream value.

Referring again to FIG. 2, at step 208, for each participant computingdevice that correspond to one of the identifiers of the subset, one ormore of the digital video streams sent to the participant computingdevice from the SFU are disabled. In an embodiment, disabling the one ormore of the digital video streams sent to the selected participantcomputing devices comprises reducing the number of streams sent to theparticipant computing device. Thus, if a particular selected participantcomputing device originally received five streams and was selected tohave one stream disabled, the SFU may stop sending one of the fivestreams to the selected participant computing device.

The disabled stream may be selected based on a rating of importance ofthe streams sent to the participant computing device. For example, inembodiments where the streams sent to a participant computing devicecomprise the most recent streams which contained above average audiolevels, the least recent of the streams to contain above average audiolevels may be dropped.

Replacing High Quality Streams with Low Quality Streams

In an embodiment, SFU 302 performs the methods for load adaptation bydisabling one or more streams in favor of lower quality streams. Thus,instead of and/or in addition to reducing the number of streams receivedby a participant computing device, SFU 302 may replace a high qualitystream with a lower quality stream. For example, the high quality streammay be a stream with a first bitrate while the low quality stream may bea stream with a second bitrate which is lower than the first bitrate.The lower bitrate may be achieved through a decrease in spatialresolution, a decrease in temporal resolution, and/or a decrease in thequality of the video signal.

In an embodiment, both the high quality stream and the low qualitystream are sent to SFU 105 simultaneously. For example, SFU 105 mayreceive simultaneous broadcast (simulcast) streams from one or more ofthe participant computing devices. For the received simulcast streams,SFU 105 may send the higher quality streams to one or more participantcomputing devices. SFU 105 may select one or more of the participantcomputing devices that are receiving the higher quality streams of thesimulcast streams in response to determining that the system load valueexceeds the particular threshold. For the selected participant computingdevices, SFU 105 may disable the one or more streams by stopping theforwarding of the higher quality stream of the simulcast streams to theselected participant computing devices and starting forwarding of thelower quality streams of the simulcast streams to the selectedparticipant computing devices.

In an embodiment, the high quality stream is compressed using scalablevideo coding such that the high quality stream contains one or moresubset streams. Scalable video coding allows SFU 105 to replace thehigher quality stream with the lower quality stream by dropping packetsfrom the transmitted stream. SFU 105 may select one or more of theparticipant computing devices using the methods described herein. Foreach of the selected participant computing devices, SFU 105 may disablethe high quality stream by dropping one or more packets from a streamcompressed with scalable video coding such that the remaining stream isof a lower quality.

For each high quality stream, SFU 105 may identify an expected decreasein load from replacing the high quality stream with a lower qualitystream. For example, if the high quality stream is sent at 4 Mbps andthe low quality stream would be sent at 2 Mbps, the expected decrease inload would be 4 Mbps−2 Mbps=2 Mbps. As both the high quality and lowquality bitrates may differ between participant computing devices, theexpected decrease in load may also differ between computing devices.

In an embodiment, SFU 105 orders streams by expected decrease in loadfor switching between a high quality version of the stream and a lowerquality version of the stream. SFU 105 may then select streams indescending order, starting with the highest expected decrease in loadand continuing until the expected decrease in load is higher than thedifference between the current load and a particular value, such as theoveruse load threshold value. By selecting streams with the highestexpected decrease in load, SFU 105 minimizes the number of streamsaffected.

SFU 105 may also compute a participant effect value for replacing highquality streams with low quality streams. In an embodiment, theparticipant effect value for each participant computing device comprisesthe quotient of the number of streams affected and the total number ofstreams. In additional embodiments, the participant effect valueincludes a value pertaining to the difference between the high qualitystream and the low quality stream. For example, the participant effectvalue may be computed as: x·q/n where x is the number of streamsaffected for the participant computing device, n is the total number ofstreams received by the participant computing device, and q is thedifference in quality between the high quality stream and the lowquality stream.

Values for q may be normalized depending on the metric used to definethe difference in streams. For example, if the difference between thehigher quality stream and the lower quality stream is the spatialresolution, then q may be computed as

$q = {1 + {C\frac{p_{h} - p_{l}}{p_{h}}}}$

where p_(h) is the pixel count of the higher resolution stream, p_(i) isthe pixel count of the lower resolution stream, and C is a normalizationconstant which is configurable to increase or decrease the effect oflarger changes in resolution. Values for q may be normalized in similarways for differences in temporal resolution and differences in videosignal quality.

Additionally and/or alternatively to the difference between the streams,the participant effect value may comprise a weighting for each streambased, at least in part, on an identified importance of the stream. Forexample, where streams are selected to be sent to participant computingdevices based on how recently the stream included greater than averageaudio levels, a first forwarded stream which included greater thanaverage audio levels most recently would have a higher identifiedimportance than a second forwarded stream which included greater thanaverage audio levels least recently. Thus, the first forwarded streammay comprise a higher weighting than the second forwarded stream. Usingthe weightings, the participant effect value for replacing a particularhigh quality stream with a low quality stream may be computed as

$\frac{x \cdot w}{n}\mspace{14mu} {or}\mspace{14mu} \frac{x \cdot q \cdot w}{n}$

where w is a computed weight based on importance of the stream.

In an embodiment, SFU 105 uses minimization techniques as describedherein to minimize the participant effect value while ensuring loadreduction past the particular load value. Additionally and/oralternatively, SFU 105 may order streams by the ratio between loadreduction and participant effect value. SFU 105 may select streams withthe highest ratio between load reduction and participant effect valueuntil the expected decrease in load is higher than the differencebetween the current load and a particular value, such as the overuseload threshold value.

In an embodiment, SFU 105 lowers stream quality for particularconference sets. SFU 105 may compute estimated load reduction for eachconference set for replacing each high quality stream with a low qualitystream and/or for replacing a particular number of high quality streamsfor each participant computing device with a low quality stream. In anembodiment, SFU 105 stores a maximum stream value and a maximum highquality stream value for a conference. Instead of reducing the maximumstream value which would remove streams for some participants, SFU 105may reduce the maximum high quality stream value for the conference,thereby reducing the quality in streams for some participants withoutremoving the streams.

SFU 105 may select conferences for quality reduction using similartechniques as selecting individual participant computing devices. Forexample, SFU 105 may order conference sets by load reduction and selectconferences with the highest load reduction until the expected decreasein load is higher than the difference between the current load and theparticular value. Additionally and/or alternatively, SFU 105 may computeparticipant effect values for conference sets and select the conferencesets that minimize the participant effect value while reducing the loadon SFU 105 until the expected decrease in load is higher than thedifference between the current load and the particular value.

Restoring Streams

In an embodiment, SFU 105 stores an underuse threshold value whichidentifies a value below which SFU 105 determines that SFU 105 isunderusing available resources. For example, an underuse load thresholdvalue for network utilization, measured as average egress bitrate on theserver over the last five seconds, may be set to 180 Mbps. Thus, if theaverage egress bitrate on the server over the last five seconds is below180 Mbps, SFU 105 may determine that the system is underusing resources.

After disabling the one or more streams, SFU 105 may be programmed orconfigured to wait a particular period of time before monitoring theload to determine whether the system load value is still above theoveruse load threshold value or whether the system load value hasdropped below the underuse load threshold value. The particular periodof time may be selected to be long enough for the system load value tochange based on the disabling of the one or more streams. If, after theparticular period of time, the system load value still exceeds theoveruse load threshold value, SFU 105 may use the techniques describedherein to select one or more participant computing devices to have oneor more streams disabled. If, after the particular period of time, SFU105 determines that the system load value is between the underuse loadthreshold value and the overuse load threshold value, SFU 105 maycontinue monitoring the system load value.

In response to determining that the current load value is below theunderuse load threshold value, SFU 105 may select one or more of thedisabled streams to be restored. In embodiments where SFU 105 stopssending one or more streams to participant computing devices, restoringthe streams may comprise starting sending the one or more of thedisabled streams to the participant computing devices. In embodimentswhere SFU 105 replaces high quality streams with low quality streams,restoring the streams may comprise replacing the low quality streamswith the previously sent high quality streams.

In an embodiment, SFU 105 selects a subset of the disabled streams to berestored. SFU 105 may select streams to restore such that the expectedload value after restoring the streams is greater than the underusethreshold value but less than the overuse threshold value. For example,SFU 105 may identify the expected increase in load of restoring streamsfor each participant computing device. SFU may select streams with thelowest expected increase in load to restore until the expected increasein load surpasses the underuse threshold value.

In an embodiment, SFU 105 may be programmed or configured to restore asmany streams as possible while keeping the expected increase in loadfrom causing the load to increase past the overuse threshold or apreconfigured intermediary value. The intermediary value may be set todecrease the probability that restoring streams causes SFU 105 tooveruse resources. For example, if the overuse threshold value is 200Mbps and the underuse threshold value is 180 Mbps, an intermediarythreshold value may be set to 190 Mbps such that SFU 105 only restoresstreams that are selected to cause the expected load to not surpass 190Mbps. Thus, if the actual increase in load of restoring the streams isslightly higher than expected, SFU 105 may still not exceed the overusethreshold value.

In an embodiment, SFU 105 may maximize a participant benefit value whilelimiting the expected load value after restoring the streams to lessthan the overuse threshold or a preconfigured intermediary value. Theparticipant benefit value for restoring streams may be computed as

$\frac{x}{n + x}$

where x is the number or streams restored and n is the current number ofstreams being received by the participant computing device and/or thecurrent number of high quality streams being received by the participantcomputing device. Additionally and/or alternatively, in embodimentswhere restoring streams comprises re-enabling high quality streams, SFU105 may be programmed or configured to compute the participant benefitscore by factoring in differences in quality and/or weightings based onimportance of streams being restored.

SFU 105 may also be programmed or configured to restore streams byconference set. For example, SFU 105 may compute, for each conferenceset, the expected increase in load of raising the maximum stream valuefor the conference set. SFU 105 may then restore streams for participantcomputing devices in conference sets with the lowest expected increasein load from raising the maximum stream value. Additionally and/oralternatively, SFU 105 may compute the participant effect value for eachconference set and raise the maximum stream value for the conferencesets that maximize participant effect value without causing the expectedincrease in load to exceed the difference between the overuse loadthreshold value or the intermediary value and the current system loadvalue.

After restoring streams, SFU 105 may be programmed or configured to waita particular period of time before monitoring the load to determinewhether the system load value is still below the underuse load thresholdvalue or has exceeded the overuse load threshold value. If the systemload value is between the underuse load threshold value and the overuseload threshold value, SFU 105 may continue to monitor the current loaduntil the system load value drops below the underuse load thresholdvalue or exceeds the overuse load threshold value. If at any point thesystem load value is below the underuse load threshold value, SFU 105may select more streams to re-enable. If the system load value is abovethe overuse load threshold value, SFU 105 may select participantcomputing devices to have one or more streams disabled.

Implementation Example—Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wired andprogram logic to implement the techniques.

For example, FIG. 4 is a block diagram that illustrates a computersystem 400 upon which an embodiment of the invention may be implemented.Computer system 400 includes a bus 402 or other communication mechanismfor communicating information, and a hardware processor 404 coupled withbus 402 for processing information. Hardware processor 404 may be, forexample, a general purpose microprocessor.

Computer system 400 also includes a main memory 406, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 402for storing information and instructions to be executed by processor404. Main memory 406 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 404. Such instructions, when stored innon-transitory storage media accessible to processor 404, rendercomputer system 400 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 400 further includes a read only memory (ROM) 408 orother static storage device coupled to bus 402 for storing staticinformation and instructions for processor 404. A storage device 410,such as a magnetic disk or optical disk, is provided and coupled to bus402 for storing information and instructions.

Computer system 400 may be coupled via bus 402 to a display 412, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 414, including alphanumeric and other keys, is coupledto bus 402 for communicating information and command selections toprocessor 404. Another type of user input device is cursor control 416,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 404 and forcontrolling cursor movement on display 412. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 400 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware andprogram logic which in combination with the computer system causes orprograms computer system 400 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 400 in response to processor 404 executing one or more sequencesof one or more instructions contained in main memory 406. Suchinstructions may be read into main memory 406 from another storagemedium, such as storage device 410. Execution of the sequences ofinstructions contained in main memory 406 causes processor 404 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and instructions that cause a machine to operationin a specific fashion. Such storage media may comprise non-volatilemedia and volatile media. Non-volatile media includes, for example,optical or magnetic disks, such as storage device 410. Volatile mediaincludes dynamic memory, such as main memory 406. Common forms ofstorage media include, for example, a floppy disk, a flexible disk, harddisk, solid state drive, magnetic tape, or any other magnetic datastorage medium, a CD-ROM, any other optical data storage medium, anyphysical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 402. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 404 for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 400 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 402. Bus 402 carries the data tomain memory 406, from which processor 404 retrieves and executes theinstructions. The instructions received by main memory 406 mayoptionally be stored on storage device 410 either before or afterexecution by processor 404.

Computer system 400 also includes a communication interface 418 coupledto bus 402. Communication interface 418 provides a two-way datacommunication coupling to a network link 420 that is connected to alocal network 422. For example, communication interface 418 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 418 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 418sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 420 typically provides data communication through one ormore networks to other data devices. For example, network link 420 mayprovide a connection through local network 422 to a host computer 424 orto data equipment operated by an Internet Service Provider (ISP) 426.ISP 426 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 428. Local network 422 and Internet 428 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 420and through communication interface 418, which carry the digital data toand from computer system 400, are example forms of transmission media.

Computer system 400 can send messages and receive data, includingprogram code, through the network(s), network link 420 and communicationinterface 418. In the Internet example, a server 430 might transmit arequested code for an application program through Internet 428, ISP 426,local network 422 and communication interface 418.

The received code may be executed by processor 404 as it is received,and stored in storage device 410, or other non-volatile storage forlater execution.

Extensions and Alternatives

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

What is claimed is:
 1. A method comprising: sending, from a selectiveforwarding unit (SFU) to each of a plurality of participant computingdevices, one or more digital video streams, wherein the SFU storesidentifiers of each participant computing device among the plurality ofparticipant computing devices; monitoring, at the SFU, a system loadvalue which estimates a load on the SFU according to any of a pluralityof different load metrics; determining that the system load valueexceeds a particular load threshold value; computing, for each of theplurality of participant computing devices, a participant effect valuewhich estimates an effect of disabling one or more streams for theparticipant computing device; selecting a subset of the identifiers ofthe plurality of participant computing devices such that an aggregatedparticipant effect value for participant computing devices correspondingto the identifiers of the subset is minimized and a particular loadreduction is obtained; for each participant computing device thatcorresponds to one of the identifiers of the subset, disabling one ormore of the digital video streams that are sent to the participantcomputing device from the SFU.
 2. The method of claim 1, furthercomprising, after disabling the one or more of the digital video streamsthat are sent to the participant computing device, sending a reducednumber of the digital video streams to that participant computingdevice.
 3. The method of claim 1, wherein the one or more of the digitalvideo streams comprise digital video streams of a first quality; themethod further comprising replacing the one or more of the digital videostreams with one or more digital video streams comprising digital videostreams of a second quality, wherein the second quality is lower thanthe first quality.
 4. The method of claim 1, further comprisingmonitoring the system load value based on network utilization of theSFU, measured as an average egress bitrate on the SFU over a particularperiod of time.
 5. The method of claim 1, further comprising monitoringthe system load value based on CPU usage of the SFU.
 6. The method ofclaim 1, further comprising selecting the subset of the plurality ofparticipant computing devices by: identifying, for each participantcomputing device of the subset of the plurality of participant computingdevices, a particular number of the one or more digital video streams todisable for the participant computing device; computing a load effectvalue which estimates an effect on the load of the SFU of disabling theone or more of the digital video streams that are sent to theparticipant computing device from the SFU, for each participantcomputing device; determining that the load effect value is equal to orgreater than the particular load reduction.
 7. The method of claim 1,wherein the particular load reduction comprises a reduction in thesystem load value that would cause the system load value to be lowerthan the particular load threshold value.
 8. The method of claim 1,further comprising: in response to disabling one or more of the digitalvideo streams that are sent to the participant computing device of thesubset of the plurality of participant computing devices, waiting aparticular period of time; after waiting the particular period of time,determining that the system load value is below a second load thresholdvalue and, in response, selecting one or more participant computingdevices of the subset of the plurality of participant computing devices;restoring one or more of the disabled digital video streams.
 9. A systemcomprising: a selective forwarding unit (SFU) comprising a memory, oneor more processors, and one or more instructions stored in the memorywhich, when executed by the one or more processors, cause: sending, fromthe SFU to each of a plurality of participant computing devices, one ormore digital video streams, wherein the SFU stores identifiers of eachparticipant computing device among the plurality of participantcomputing devices; monitoring, at the SFU, a system load value whichestimates a load on the SFU according to any of a plurality of differentload metrics; determining that the system load value exceeds aparticular load threshold value; computing, for each of the plurality ofparticipant computing devices, a participant effect value whichestimates an effect of disabling one or more streams for the participantcomputing device; selecting a subset of the identifiers of the pluralityof participant computing devices such that an aggregated participanteffect value for participant computing devices corresponding to theidentifiers of the subset is minimized and a particular load reductionis obtained; for each participant computing device that corresponds toone of the identifiers of the subset, disabling one or more of thedigital video streams that are sent to the participant computing devicefrom the SFU.
 10. The SFU of claim 9, wherein the one or moreinstruction, when executed by the one or more processors, further cause,after disabling the one or more of the digital video streams that aresent to the participant computing device, sending a reduced number ofthe digital video streams to that participant computing device.
 11. TheSFU of claim 9, wherein the one or more of the digital video streamscomprise digital video streams of a first quality; wherein the one ormore instruction, when executed by the one or more processors, furthercause replacing the one or more of the digital video streams with one ormore digital video streams comprising digital video streams of a secondquality, wherein the second quality is lower than the first quality. 12.The SFU of claim 9, wherein the one or more instruction, when executedby the one or more processors, further cause monitoring the system loadvalue based on network utilization of the SFU, measured as an averageegress bitrate on the SFU over a particular period of time.
 13. The SFUof claim 9, wherein the one or more instruction, when executed by theone or more processors, further cause monitoring the system load valuebased on CPU usage of the SFU.
 14. The SFU of claim 9, wherein the oneor more instruction, when executed by the one or more processors,further cause selecting the subset of the plurality of participantcomputing devices by: identifying, for each participant computing deviceof the subset of the plurality of participant computing devices, aparticular number of the one or more digital video streams to disablefor the participant computing device; computing a load effect valuewhich estimates an effect on the load of the SFU of disabling the one ormore of the digital video streams that are sent to the participantcomputing device from the SFU, for each participant computing device;determining that the load effect value is equal to or greater than theparticular load reduction.
 15. The SFU of claim 9, wherein theparticular load reduction comprises a reduction in the system load valuethat would cause the system load value to be lower than the particularload threshold value.
 16. The SFU of claim 9, wherein the one or moreinstruction, when executed by the one or more processors, further cause:in response to disabling one or more of the digital video streams thatare sent to the participant computing device of the subset of theplurality of participant computing devices, waiting a particular periodof time; after waiting the particular period of time, determining thatthe system load value is below a second load threshold value and, inresponse, selecting one or more participant computing devices of thesubset of the plurality of participant computing devices; restoring oneor more of the disabled digital video streams.